SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1
V-Mail: 3D-Enabled Correspondence
about Spatial Data on (Almost) All Your Devices
Jung Who Nam , Tobias Isenberg , and Daniel F. Keefe , Senior Member, IEEE
Abstract—We present V-Mail, a framework of cross-platform applications, interactive techniques, and communication protocols for
improved multi-person correspondence about spatial 3D datasets. Inspired by the daily use of e-mail, V-Mail seeks to enable a similar
style of rapid, multi-person communication accessible on any device; however, it aims to do this in the new context of spatial 3D
communication, where limited access to 3D graphics hardware typically prevents such communication. The approach integrates visual
data storytelling with data exploration, spatial annotations, and animated transitions. V-Mail “data stories” are exported in a standard
video file format to establish a common baseline level of access on (almost) any device. The V-Mail framework also includes a series of
complementary client applications and plugins that enable different degrees of story co-authoring and data exploration, adjusted
automatically to match the capabilities of various devices. A lightweight, phone-based V-Mail app makes it possible to annotate data by
adding captions to the video. These spatial annotations are then immediately accessible to team members running high-end 3D graphics
visualization systems that also include a V-Mail client, implemented as a plugin. Results and evaluation from applying V-Mail to assist
communication within an interdisciplinary science team studying Antarctic ice sheets confirm the utility of the asynchronous,
cross-platform collaborative framework while also highlighting some current limitations and opportunities for future work.
Index Terms—Human-computer interaction, visualization of scientific 3D data, communication, storytelling, immersive analytics.
F
1 INTRODUCTION
S
PATIAL data visualization has long been essential for
breakthroughs in medicine, science, engineering, the
humanities, and more (e. g., [1], [6], [18], [41]). Sharing
these visualizations with others, however, remains a major
challenge for many 3D, scientific, and/or big data visu-
alization applications because they typically require high-
end devices and data processing—the type of computing
infrastructure that is not portable and not accessible to most
non-experts. One strategy for addressing this problem is
to port the exploratory visualization tools scientists use on
their current high-end visualization systems (e. g., advanced
volume rendering) to the commodity devices that people
use regularly, such as through remote rendering [26] or
new web technologies [29], [38]. A fundamental limitation,
however, is that scientific visualization applications are often
complex, requiring substantial expertise on the data and
on how to use software features. Thus, even when it is
possible for applications to run on commodity devices, there
are still serious challenges to overcome in order to engage
new audiences (e. g., people who may have little incentive
to explore the data) and, often, even our team-science
collaborators, who take on advisory roles in developing
visualization techniques.
In light of this situation, we take a different, comple-
mentary approach. We argue that bringing full-fetched
data exploration capabilities to every device may not be
the most important requirement of high-end collaborative
Jung Who Nam and Daniel F. Keefe are with the University of Minnesota,
USA. E-mail: { namxx054 | dfk }@umn.edu .
Tobias Isenberg is with Universit´e Paris-Saclay, CNRS, Inria, LISN, France.
E-mail: tobias.isenberg@inria.fr .
Manuscript received December 5, 2022.
visualization for spatial 3D data. Instead, what is fundamen-
tally needed is to improve human-to-human communication
about the data, and the way to do that is to make this
communication work on everyone’s existing devices. This
approach conceptually builds closely on prior work in
storytelling with data visualizations [8], [19], [31], which has
been explored extensively for 2D abstract data but less so for
spatial 3D data. By adopting a keyframe-based storytelling
metaphor [3], [40], we have developed a system that creates
animated data stories from data views captured directly
from exploratory visualizations along with a mechanism of
using an URL of a standard video file as the base ‘token’
for distributing a data story. By packing the communication
about the data into a standard video format and sending
this critical human-to-human communication rather than the
raw data and visualization application, our system leverages
existing e-mail and social media applications as tools for
distributing stories.
Many forms of storytelling are, however, collaborative
activities by nature [20], [23], [42], and people often take on
different roles in this process [7], [11], [23]. We, thus, also
support tasks that data story audiences may carry out after
an initial viewing of a story such as providing feedback to
the original author, verifying claims made in the story, or
even adding new content to the story.
To turn data storytelling into a collaborative and iterative
process, we extend our system and introduce the V(isualiza-
tion)-Mail framework. The key approach is flexible export
of data story where a single V-Mail story token can trigger
experiences with different storytelling fidelities, based on
the absence or presence of dedicated software (Fig. 1). By
having options from simply viewing the story on a video
player to opening it within specialized visualization software
to explore and annotate the 3D data, it is possible to cover
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2
Fig. 1. Three working platforms that demonstrate different storytelling fidelities of V-Mail for analyzing Antarctic ice-sheet data: (right) anyone with a
video player can at least passively view the story, including annotations made by others; (left) in the highest-fidelity case, the story unlocks data on a
V-Mail server than can be loaded via a plugin for desktop-based visualization applications, where users can explore and annotate the 3D data more
deeply; (middle) the mobile client works as a custom video player with mechanisms for interactively viewing the story and adding annotations.
a wide range of use-cases for collaborative discussion and
3D data analysis, including making it possible for users with
different tasks and levels of familiarity with visualization
applications to work in unison.
We also present several extensions to this core idea to
further enhance 3D-specific data storytelling. For example,
since understanding the spatial context for specific data
views is often critically important, the views are represented
as 3D frames floating in the same space as the data visu-
alization, enabling the audience to see them from various
vintage points. In addition, to increase clarity, we employ a
staged animation when the transition between data views
would require changes to several visualization parameters,
for example, when new data filters or color maps have been
applied.
In summary, the main contributions of this article are:
the design of a novel visualization framework to fa-
cilitate 3D-enabled multi-person correspondence about
spatial datasets for team science,
an extension of keyframe-based storytelling metaphors
to support co-authorship, treating the created stories as
collaborative documents authored asynchronously,
the design of a data story server and communication
protocol for using V-Mail across an ecosystem of hard-
ware platforms that seamlessly adapt the fidelity of the
experience to the hardware, and
user feedback and lessons learned from a first evaluation
of using V-Mail in a real team-science setting where team
members utilized all V-Mail client platforms we created
(desktop app, mobile app, and native video players on
multiple mobile and web-based devices).
2 RELATED WORK
V-Mail relates to research on interactive tools, techniques, and
formats for storytelling with data visualization and builds
upon multiple topics (e. g., animation and annotation, inter-
activity, co-authorship and collaboration) that have become
consistent themes in this research. We note that, although
storytelling techniques have been used for presenting both
spatial [27], [28], [36], [40] and non-spatial [4], [10], [22], [23],
[32], [34] data, recent work has focused more consistently on
non-spatial data. V-Mail, therefore, builds upon a trend of
customizing storytelling support tools for specific types of
data (e. g., [3], [25], [35], [40]), but does so for the specific case
of spatial 3D data, as often encountered in domains such as
physics, astronomy, biology, and medicine.
2.1 Co-Authoring and Interacting with Stories
Many forms of storytelling are, by nature, collaborative
activities [20], [23], [42] and people often take on different
roles in this process [7], [11], [23]. In a recent survey, Tong
et al. [36] describe authoring tools for parallel storytelling,
where multiple authors work simultaneously to create the
final visualization content. Like V-Mail, this type of story-
telling blurs the lines between storyteller and audience; the
audience partakes in the storytelling and the initial storyteller
is also an audience member. In sense.us [13], for example,
users communicate collaboratively using annotations and
comments posted online on web-based data representations.
The audience also partakes in the storytelling process in
GeoTime [9]. In an accompanying text editor, authors write
text-based data stories using bookmarked data views as
references to claims. By scrolling the editor filled with
the links, the audience has the ability not only to see the
message with text descriptions but also to flip through
saved visualization states for further exploration. Further,
by modifying text in the editor, the audience can add new
content or even new data views found in their own data
analysis sessions.
V-Mail is similar to GeoTime in that multiple users
collaborate on visualization presentations. Unlike GeoTime,
however, V-Mail employs several simple yet effective tech-
niques to convey different users’ contributions. For example,
as in the work by Heer, Viegas, et al. [13], [39], each comment
shows the author of the edit. Further, V-Mail incorporates
lower-fidelity options for viewing and collaborating with
others on low-cost devices such as mobile phones. By sharing
a visualization state via a URL [13], [39], V-Mails can be refer-
enced in other communication media such as personal blogs.
Similarly, by using video files as the means of distribution
[14], V-Mails can be widely distributed, including through
social media. V-Mails provide users with different options for
consuming data stories, from passive viewing to interacting
with them, as there is no one correct level of interactivity
on the spectrum for constructing a narrative visualization
[11], [28], [34]. While interaction can distract viewers from
an initial story arc [20], the ability to interactively view data
stories is typically seen as giving the audience a feeling of
involvement and lends credibility [21], [27], [28], [40], [42].
Existing web-based systems that rely on remote rendering
[26] or new web technologies [29], [38] can also allow people
to carry out visualization tasks on the commodity devices
they use regularly. These systems, however, often target data
exploration tasks, whereas we focus on data communication
with the support for collaborative annotation and storytelling.
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 3
Also, our first prototype provides storytelling techniques on
top of visual outputs from exploratory visualization systems,
so the V-Mail concept could be applied to visualization
software built as standalone applications.
2.2 Annotating Spatial 3D Data
When compared to creating data stories for 2D or non-
spatial data, authoring stories about 3D spatial data includes
another special consideration in that it requires connecting
text, speech, drawings, or other input from authors to specific
spatial views or data elements. These software features are
often described as 3D annotations, which can be tied directly
to data objects [37], [40]. Alternatively, graphical annotations
can be drawn in ‘screen-space’ [3], [13], [25], [37]. Without
the need to explicitly reference data elements, this ‘screen-
space’ drawing with ‘free-form’ annotations enables users to
be more expressive [13] and even to draw in ‘empty space,’
which can be useful in sparse 3D scenes [37]. However,
free-form ‘screen-space’ annotations are typically not useful
when viewed from other vantage points, as the spatial
context is only valid from the exact viewpoint where the
annotations were made. Some form of navigation between
annotations is needed along with an ability to see what
viewpoints include annotations. V-Mail thus implements
screen-space annotations in the style of Boom Chameleon
[37]. After creating an annotation from one viewpoint, we
add a snapshot of the annotated view to the scene as a 3D
widget (view frustum with a textured film plane) that can be
seen from other vantage points and returned to later.
2.3 Arranging and Transitioning Between Multiple Views
Many data storytelling systems provide multiple comple-
mentary data views, and they are considered critical for
3D data storytelling [3], [36], [40]. One reason is that
understanding the complex spatial relationships in 3D data
often requires multiple and/or animated perspectives [20],
[24], [28], [34]. By smoothly transitioning between these
perspectives, well-designed animations can allow the audi-
ence to build a congruent mental model of the spatial data
[10]. Different mechanisms exist for authoring animations
for data storytelling. These can be as simple as recording a
sequence of interactions performed by the storyteller [40]
or providing animation templates to assist users in creating
a final animation [3]. Authoring a meaningful animation,
however, can be complicated. Automated techniques are thus
also useful, especially in time-critical scenarios such as urgent
care in a hospital [14] and even for science dissemination [21].
V-Mail, adopts a semi-automatic approach, similar to that of
Lidal et al. [25]; the animations are created by interpolating
the values the storyteller has captured/created in discrete
states. We extend this concept by utilizing a staged animation
[3], [12] to increase clarity when the transition would require
changes to several parameters at once. Additionally, V-Mail
includes a linked 2D storyboard view to define the temporal
context for each key state and to re-sequence the story if
desired. This feature is motivated by prior work that created
guided 3D visualizations by capturing and ordering multiple
views and found that breaking the presentation into steps
can improve comprehensibility [40].
3 D ESIGN CRITERIA AND CORE DE SIGN CONCEPTS
Based on this discussion of existing means for communicat-
ing about or using spatial 3D data and the challenges we
outlined in the previous sections, we now translate these
points into specific design criteria and introduce the design
concepts of V-Mail that guided us toward our vision.
3.1 Design Criteria
Our overall approach is driven by today’s huge spectrum
of data communication possibilities, yet with no solution
that matches our vision. On one end of the spectrum,
people use e-mails enriched with screenshots from general-
purpose visualization tools such as ParaView [2]. This has
the advantage that one does not have to worry about the
device of the addressee, yet messages are limited to texts
with few images, with little expressiveness. Video exports can
also be used to showcase visual representations with higher
fidelity and in a continuous fashion, yet this only supports
one-way communication. Moreover, communication features
such as drawing and commenting are typically missing in
e-mail based communication. The other end of the spectrum
is marked by storytelling tools, such as that by Wohlfart
and Hauser [40], in which recipients can load a shared
visualization not only to view but also to explore. Unfor-
tunately, such tools require collaborators to have a similar
hardware and software setup and often hours of training
to use the visualization application. As argued previously,
this often makes them accessible only to experts. Our key
goal is thus to facilitate communication about spatial 3D data
without having to worry about recipients’ devices and skill
levels, yet with an ease of communication, responding, and
commenting that approaches that of e-mail. To achieve this
vision, we formulated the following design criteria, which
are closely informed by ongoing data visualization research
on storytelling, communication, and collaboration:
As audiences have different goals and needs,
C1
we need to provide them with different options for
consuming the presented material, e. g., with different
levels of interaction and detail [11], [23], [42] and
C2
we want to support a range of devices to view messages
and provide feedback so users can pick a visualization
suitable for their tasks and current situations [15].
As there can be multiple ways of interpreting results,
C3
messages should capture the thought process that lead
to a conclusion so others can make informed decisions
[25], [30], [40]
C4
and, when going through the steps of presenting spatial
data, it is important to a method should be provided for
the audience to maintain spatial correspondence as the
presentation proceeds and to understand where, spatially
within the data, each step takes place [21], [28], [36], [37],
C5
presenters should thus have various options to express
their ideas and thoughts and to draw the audience’s
attention to particular viewpoints and data features [25],
and
C6
the audience should have the ability to add to and edit
the content so that groups can exchange ideas back
and forth iteratively as they work collaboratively to
understand and interpret the impact of the data [23].
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 4
3.2 V-Mail Core Metaphor and Design Concepts
The core metaphor behind V-Mail is that of a book—
specifically, we recall children’s’ picture books that include
text and imagery on each page and are often organized so
that each page functions as a mini “episode” of the story. A
child looks at a page in a book and reads the text at their
own pace to understand the setting and characters pictured.
Then, to advance, the child flips to the next page to continue
to the next episode. Books for adult readers follow a similar
progression of steps, but this is often arranged at the level of
chapters. The relationship of one chapter to the next may not
be immediately apparent, but the perspective provided by
each chapter is, ultimately, critical to the interpretation of the
whole book. Books of all forms naturally lend themselves to
a variety of annotations, from hand-drawn illustrations to
remarks scribbled in the margins.
In V-Mail’s approach to storytelling with spatial 3D data,
a ‘page’ is a
view of the 3D dataset
that shows a particular
situation. Annotations can be provided by
sketching
onto
the 2D projection of the 3D view or by attaching textual
annotations to the view. ‘Flipping to the next page’
animates
from one 3D view to the next
, allowing the viewer to follow
along while maintaining spatial context. Several page flips
can be connected into
chapters
’, but the transition from
the end of one chapter to the beginning of the next can be
more abrupt (i.e., without an animated transition). As in the
children’s picture book, it is possible to animate
forward and
backward
through the data story, with
different playback
modes
to control the time with precision or to quickly skip
to a new chapter. With this approach, complex messages are
broken down into more digestible, visual, and expressive
forms (C3–C5).
The V-Mail frameworks requires just three additional con-
cepts to be added to this core “book metaphor functionality:
Floating 3D pages
—When presenting visualization re-
sults with a set of captured data views, we often think of
a panel listing the views; e. g., thumbnails on the viewer in
AniViz [3]. V-Mail includes such a panel, but in addition
to this panel, the data views are also included directly
within the spatial visualization of the underlying 3D data.
Together with the smooth transitions between the data views
achieved with animations, these floating pages assist users
in understand the spatial correspondences between views as
they progress through a data story (C4).
Living documents
—To remove the one-way character
of existing 3D data communication and turn the data
communication into a collaborative and iterative process
accessible to non-experts, V-Mail treats the data stories as
living documents (C1, C6). The framework makes it possible
for users to explore a V-Mail, provide annotations, and extend
stories in a way that scales with the capabilities of the device
being used.
Communication via a
lightweight token
—To realize this
device flexibility (C2) and thus be able to engage a broad
audience (C1), V-Mail does not rely on dedicated rendering
support. Instead, stories are shared as a video URL—online
token that can be interpreted based on the used device
and the absence or presence of dedicated software. The
video-based, low-fidelity version of the data story makes it
possible for users to open it on a mobile phone with a video
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2. The V-Mail desktop client allows users to create a story directly
from a visualization application: (a) the top portion is the visualization
application; (b) the storyboard panel is placed on the bottom; (c) captured
data views in a story; (d) the timeline interface provides playback options
and indicates the current time in a story; (e) the drop-down menu provides
options for opening, uploading, and sharing a story; (f) the comment panel
shows comments on the current data view.
player, for instance while traveling. However, this lowest
level of fidelity supports only simple viewing options (e.g.,
the web client in Table 1). A range of higher-fidelity versions
that require dedicated software offer additional exploration
and annotation capabilities. On a high-end desktop, users
can freely explore the data depicted in the story and add
additional pages or chapters (e.g., the desktop client in
Table 1). A middle-fidelity version only supports exploration
at certain view locations but with fairly advanced annotation
capabilities and story navigation features (e.g., the mobile
client in Table 1). To make it easy to manage such device
flexibility, V-Mail associates all story data with the video, and
shares it as a URL token that is opened appropriately by the
different devices.
4 DETAILED DESIGN AND IMPLEMENTATION
We now describe the visual components, interaction tech-
niques, and distribution mechanisms required to realize this
vision.
4.1 Visual Components
A V-Mail story is composed of data views with drawings
and comments from different users. We add animations
in-between the data views to convey their spatial correspon-
dences in a continuous fashion. Our V-Mail desktop client is
one example of a V-Mail client (Section 4.3.3) and is a plugin
to an exploratory visualization application [17]. The plugin
is comprised of three visual components: (1) the storyboard,
(2) floating 3D data views, and (3) animated-transitions.
4.1.1 The Storyboard with the Comment Panel
This first visual component shows an overview of a data
story and the input from contributors (Fig. 2(b)). Thumbnails
of data views provide a quick glance at important points in
a story and its presentation order (Fig. 2(c)). We also overlay
drawings on the thumbnails to indicate visual highlights from
users. Sometimes it may be more appropriate to textually
describe what is shown in a shared data view; e. g., for
writing a case number in a data ensemble or the name of a
rendering algorithm. The comment panel provides an interface
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 5
(a) (b)
Fig. 3. Example usages of 3D data views in data communication: (a) the
user talks about data features focused in the views; (b) a bird’s eye view
shows the locations being discussed in the data.
for leaving such notes and viewing comments left by others
(Fig. 2(f)); each comment includes the author’s name.
4.1.2 Floating 3D Data Views
This second visual component shows the user how the
story is spatially located in the 3D data space. As shown
in Fig. 3 and inspired by Tsang et al. [37], captured data
views are rendered in the same space as the underlying data
visualization so that they are visible from other vantage
points. If a data view is captured close to a feature of
interest, the user can zoom out and see a representation
of that viewpoint in the context of a wider angle view of
the data. By making this spatial information visually explicit,
users can also visually reference other data views from the
current view: a shot can show adjoining data views to convey
their spatial relationship, before transitioning to each view
(Fig. 3(a)). A spatial overview of all of the activities in a data
analysis session can also be easily created by showing the
data views scattered around visualization, from a bird’s-eye
perspective (Fig. 3(b)).
4.1.3 Animated Transitions
Animated transitions help to convey the spatial and contex-
tual correspondences between data views in a story. Instead
of jumping from one data view to the next, we use animated
transitions that the audience can gradually adapt to the visual
changes in different data views. The animated transitions are
created automatically whenever a new data view is added
to minimize the time and cognitive effort needed to create a
V-Mail story.
Four types of animated transitions (Fig. 4) can be used,
depending on the differences between the viewpoints and
visualization states in the adjoining data views. If only the
camera location changes, the animation (1) translates the
camera from a starting location to the next location.
1
When
only the visualization state has changed, the animation (2)
gradually changes the values of visualization parameters
from the previous state to the next by interpolating the
values of just the parameters that have changed. Users
may stop the transition to explore the visualization in an
intermediate frame, but this exploration is only possible
when the set of visualization parameters of two states are
the same and the parameters can be changed continuously
(e. g., changing a slicing plane in volume rendering). In
cases where there are no obvious intermediate frames (e. g.,
changing a visualization type or dataset) we (3) use an image-
based blending approach. This blends two images where
the screenshot from the previous data view fades-out and
gradually fades-in into that of the next data view (see the
second row of Fig. 4). When both camera location and state
change, we (4) break the transition into two stages. First the
camera moves to the next location. Then, the visualization
state gradually changes to that of the new data view (see the
last row of Fig. 4).
1.
In our prototype, a linear interpolation is used; however, other
motions that optimize camera motion to support specific goals could
be useful. We thus implemented the animated transitions as abstract
methods to enable future developers to override default animations
with alternatives.
cam. vis. type animated transitions
6= = 1
=======================================================
move to the next location
= 6=
2
or
3
=======================================================
interpolate to the next visualization state by parameter (type 2) or through blending (type 3)
6= 6= 4
==========================
move to the next location
==========================
interpolate to the next visualization state
Fig. 4. Different animations to transition from one data view to another. Each type shows the changes in-between two adjacent data views.
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 6
Fig. 5. The user sketches and leaves comments on the current view in
the annotation mode. A drawing pallet is for specifying the color and the
size of a brush. Clicking the toggle on the top of the storyboard switches
the current mode to the exploration mode.
4.2 Basic Story Interaction
We introduce several interaction techniques to create, view,
and explore V-Mails.
4.2.1 Creating a story and annotating data views
We begin creating a story with an empty storyboard. Users
can navigate in the data visualization application (Fig. 2(a))
using typical 3D view manipulation techniques, and then
add a data view to the storyboard via a dedicated button
(“+” in Fig. 2(b)). We then capture the view’s thumbnail and
insert it as a reference into the storyboard, also extending
the timeline (Fig. 2(d)) appropriately (for
2 thumbnails)
to account for the needed transition animations. Users can
also re-arrange the data view sequence using simple drag-
and-drop interaction, and they can remove data views as
desired.
To be able to tell a story effectively or to respond to one,
it is also often useful to sketch directly on a view to direct
someone’s attention to particular features, which we support
on captured data views (Fig. 5). To avoid conflicts between
sketching and view manipulation, we differentiate explicitly
between these two modes and users can switch between
them using two buttons at the bottom of the data view. In
addition to the sketches, longer textual comments can be
typed and associated with the data views (Fig. 2(f)).
4.2.2 Uploading a story
When ready to share the V-Mail, users upload the story to a
server. Instead of uploading a story to the server whenever
there is a change, we initiate this upload process explicitly
(via an entry in the drop-down menu). We chose this design
to enable the user to ideate and draft a story before sharing
it to others, similar to drafting an e-mail before sending it.
4.2.3 Viewing an animated data story
As a sequence of data views with animated transitions, a
V-Mail story is akin to a video with chapters. We thus borrow
different playback modes from traditional video players and
adjust them to our application (Fig. 6).
As in normal video players, playback continues until
the end of the story (top row in Fig. 6). However, different
from normal videos, the V-Mail player pauses briefly at each
‘chapter mark’ to devote additional time to and indicate the
Data View #1 Data View #2 Data View #3
Fig. 6. Playback options & corresponding buttons. Top: plays the anima-
tion from start to finish, or in reverse; middle: plays it until it reaches the
next/previous data view; bottom: jumps to the next/previous data view.
(a) (b)
Fig. 7. The data visualization application and the storyboard are visually
and interactively linked. (a) Hovering over a 3D data view frame in the
3D view (the center one here) highlights the frame and its corresponding
thumbnail on the storyboard; (b) clicking on a data view frame in the 3D
view makes the storyboard jump to this data view.
significance of these key perspectives. To further emphasize
the data views, the player also includes mode where playback
stops completely at each data view so users may read
annotations in detail (second row in Fig. 6) and respond
to them—similar to the playback style used by Wohlfahrt
and Hauser [40]. Finally, for faster navigation, the player also
supports jumping between data views without animations
(bottom row in Fig. 6).
2
Another aspect of data stories that
is different from normal videos is that it can be useful to
view data stories, especially the transitions between views,
in reverse. Thus, all three of the play modes include support
for reverse playback.
To allow users to explore V-Mails in a more fine-grained
manner, similar to video players, we also provide a time-
line slider (Fig. 2(d)). To emphasize the aspect of chapter
transitions we constrain the timeline dragging to between
two neighboring data views, so that it is easy for a user to
focus on the visualization view while controlling the timeline
with a mouse. This constraint is reset on each mouse down
(for exploring a new transition) and can be removed with a
modifier key.
4.2.4 Linking free exploration and storyboard timeline
Again different from traditional video players, the interaction
in a data story is not restricted to just playing back a pre-
created video stream. Instead, a user may exploring the
3D data visualization freely, potentially to capture slightly
different shots or to better understand a feature in the data.
After such a free exploration (and if no new shot was added
to a timeline), the viewer may want to transition back to
the timeline. We support both of these types of transitions
by keeping the 3D visualization view and the timeline
exploration linked and use animation transitions from the
set described in Section 4.1.3. Each time a user switches
2.
The desktop client does not have these skip buttons because the
skipping navigation can be done by clicking thumbnails (Fig. 2(c)). In
the mobile client, however, we place the buttons in the bottom corners
of the display to provide easier access (Fig. 9).
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 7
(a)
(b) (c)
Fig. 8. A story URL works as a lightweight token for sharing and opening
the story. (a) our V-Mail clients show several stories with their URL;
(b) after a story’s upload, the client shows the URL; (c) opening the URL
takes the user to a webpage where the story video can be played and
the story can be opened in a dedicated client via a button (bottom).
from timeline to 3D view, we store the current position in
the timeline to be able to transition back. In the opposite
direction, we create a transition from a current state in the
3D viewer to the last active one. We also allow users to
jump to stored data views, not only by interacting with the
storyboard view, but also by clicking on the respective data
view frames in the 3D view (Fig. 7).
4.3 Cross-platform Collaboration Mechanisms
To truly support our cross-device collaboration vision (C2)
we cannot rely on all users having access to a high-end PC,
which may be required for the style of interection described
this far. We thus now describe how users respond to V-
Mails on different devices and introduce the platform-specific
prototypes (i. e., V-Mail clients) that we have developed so
far.
4.3.1 Flexible export of data story
As we want to support a wide ecology of devices, we have
to ensure that they can show a V-Mail without advanced
rendering hardware and even potentially without a dedicated
client. As a most basic representation we thus rely on a
traditional video that we generate from the data story, just as
if one would have used a normal playback to watch it. Any
device that supports video file playback (we use MP4 files)
can thus show this most basic V-Mail format—including
virtually all mobile phones, tablets, and laptops. Devices or
players that also support video subtitles embedded in video
files can also show the textual comments. Similarly, we also
embed chapter marks for those video players that support it.
If a person receives V-Mails on a more regular basis, they
can decide to install a dedicated tool that can enable them to
further engage with V-Mail. To support additional features,
we export a collection of files for each story in addition to the
video file (Table 2 in Section 5). In this case we provide access
to the dedicated play modes we described in Section 4.2.3,
and for this purpose we also include a video file exported
in the reverse direction (video backward.mp4). To allow users
to adjust or add to the graphical and textual comments, we
save both types as separate files (ann i.png, msg.json), that
can be updated by the dedicated player after a change. In
addition, we provide some simple ‘3D exploration’ on low-
power devices. As we cannot rely on rendering hardware, we
use 360
°
images that we create during story export for each
of the data views (img360 i.png). We also store all data views,
camera locations, and visualization states in the msg.json
(a)
(b)
(c)
Fig. 9. We created the V-Mail mobile client (a) as a custom video player
with visual components and mechanisms for interactively viewing and
editing a story; clicking on the top-left share button in the viewer copies
the story’s URL and shows messaging apps to share the URL with (b);
when clicking an e-mail app, we create a message with the name of the
story as a title and the story’s URL pasted into the main body (c).
file for users with sufficient high-end hardware and the full
visualization software. Only with this data ix ig possible
to adjust the data views, add new ones, rearrange their
sequence, and re-export an updated story.
4.3.2 URL as a lightweight token for story sharing
While we export a collection of files for each story (Table 2),
we do not want to bother users with having to manage and
send these files. We thus manage the file collection on a
dedicated server (that only has to provide file management
and remote access) and instead share a story in form of
a URL that points to the repository. An essential benefit
of using a URL is that this allows users to not only link
to a story in an e-mail but also to share a story on social
media. We create this URL after each export (Fig. 8(b)). When
opened in a web browser, the URL simply shows a video
player (Fig. 8(c)). On devices with access to our dedicated
clients the website, the URL provides a means to open the
story in the dedicated software. Alternatively, the clients also
provide a list of stories stored on the server (Fig. 8(a)). In
both cases, the clients then access the files needed for their
specific capabilities and provide the respective interaction
tools, as we detail next.
4.3.3 A device ecology for collaborating on data stories
We implemented three specific clients as a proof of concept
and to mirror the data levels we described in Section 4.3.1.
In Fig. 1 we show photos of these solutions and in Table 1
we compare them by the features they support.
As just described, our most basic
web client
relies on
the video player available on virtually all connected devices
(Fig. 8(c)) to facilitate passive storytelling [40]. Depending
on the functionality of this player, one may only perceive the
story as a simple playback with graphical annotations for the
data views. If the player supports it, however, one may also
jump between chapters and/or see the textual comments
that are available as video subtitles.
The web client can be accessed on virtually any connected
device and uses the device’s built-in video playback. With
it we can thus reach completely new audiences with data
stories. Even if someone never used a data story before and
thus does not have or is not willing to install dedicated
software, they can still view them. Depending on the media
channel used to share the URL, they can also respond to a
story such as by commenting in a separate e-mail. Moreover,
this form of consuming also allows users to explore the story
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 8
TABLE 1
Availability and fidelity of operations that facilitate collaboration around shared stories and their prerequisites, by client/platform.
web client: traditional video player mobile client: custom video player desktop client: plugin to vis. tool
interactive storytelling
story navigation
playback and timeline controls with-
out pausing points
playback and timeline controls with pausing points,
storytelling affordance that distinguishes story nodes and transitions
exploration no support
(i. e., passive story-
telling [40])
partially
(360
°
views; i. e., storytelling
with interactive approval [40])
full support
(all four modes of author-
and reader-driven storytelling [40])
collaboration around shared stories
view comments partially (if player shows subtitles) yes yes
view annotations yes yes yes
leave comments no support yes yes
add annotations no support yes yes
extend story no support no support yes (add, remove, rearrange story)
software and hardware prerequisites
software needs none dedicated app special visualization software
hardware needs virtually none virtually none high-end GPU & CPU processing
on their own first [23], before, potentially, adding to it and
working collaboratively with other authors.
To explore the situation where users are willing to install
a dedicated tool, such as a
dedicated mobile app
on their
phone or tablet, we developed an Android V-Mail client app
(Fig. 9), which makes possible a higher fidelity of interaction
for both phones and tablets. On top of playing back the video
of the data stories, the app also allows users to make use of
our advanced playback modes, add new annotations (both
graphical and textual), and have limited 3D exploration via
the pre-computed 360
°
images for each data view. As it is
likely used on small-screen devices, the app does not support
changing the chapter order, and instead of using thumbnails
for the data views in the storyboard, the app places crumbs
on the timeline to provide visual and navigational support
to users.
The mobile app facilitates a much richer form of col-
laboration by means of data stories than the web client,
including being able to directly comment within the viewing
environment and some 3D exploration. However, since this
functionality relies only on the built-in video playback avail-
able in virtually all devices, the processing power required
is essentially the same as the web client. Also, because it
runs on mobile devices, it inherently allows people to engage
with data stories like they do with other online media, for
example, during a break, at home, or while traveling. In fact,
the URL format makes it possible for people to share V-Mails
on a variety of communication platforms, from e-mail to
messaging apps, already installed on their phones.
For full-fidelity interaction, however, we created a
desk-
top V-Mail plugin
to a data visualization application [17]
for both Windows and MacOS. This setup provides the
functionality described in Section 4.2. It requires a high-
performance computer, but it provides full flexibility in
data exploration, story creation, and editing. At least one
desktop client like this is needed to create an initial sharable
data story, and this high-end client also makes sure that
any updates from mobile app users get re-embedded into
the video exports. This client also supports creating story
branches with new sharable URLs.
Video Renderer
360° Images Viewer
Exploratory
Visualization
Software
DB
Storyline Edits
Annotations
Story Navigation
Desktop Plugin
File
Storage
V-Mail Server
Data
Annotations
Story Navigation
Mobile Client
HTML <video>
Web Client
Fig. 10. System architecture. V-Mail is composed of a server and three
visualization clients. The desktop plugin is connected to an exploratory
visualization software. The mobile and web clients support a portion of
data storytelling activities around visual outputs stored in the server.
5 SYSTEM AND IMPLEMENTATION NOTES
Our current V-Mail implementation comprises three clients
(Fig. 10). We built the desktop plugin on top of an exploratory
visualization software; it provides all the storytelling capa-
bilities. When the user of the plugin uploads a story, files
shown in Table 2 are generated and uploaded to the V-Mail
server. One of these files is a video, and the mobile and web
clients work with this video file. The mobile client is a special
video player for data videos with the dedicated play modes
we described in Section 4.2.3 and annotation features. The
web client relies on a video player within a browser.
5.1 V-Mail server and general operation
The V-Mail server comprises a database and a file storage.
The purpose of integrating the database is to check available
stories without opening files in the storage. It has a single
table; each row represents a V-Mail story, and the columns
are: an unique ID of the story, a name given by the user who
created it, and last times modified by each client. The file
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 9
TABLE 2
Files in a V-Mail story folder
File Use by Description
msg.json Desktop, Mobile Contains information about camera views, visualization states, and comments on data views.
video.mp4 Mobile, Web Shows the animation as a video file.
video backward.mp4 Mobile Shows the animation in a reverse direction, and is used to provide a smooth backward play.
per data view
img i.png Desktop A snapshot of the current view, that is shown as a thumbnail for the data view
nextImg i.png Desktop
A snapshot captured from a camera location of the next data view, that is used to support the
fourth type of animated transition
annImg i.png Desktop, Mobile An image of drawings
img360 i.png Mobile A 360° image of the view, that is used to provide limited 3D exploration
for the web-client
index.html Web The default page
subtitle.vtt Web Shows comments as a subtitle for the video.
playlist.m3u Web Provides data view portions as chapters in the video.
storage contains additional files for V-Mail stories (Table 2).
Files for each story are saved in separate folders, and names
of the folders are the unique IDs listed in the database.
When a user opens the desktop or mobile client, the client
accesses available stories from the database and shows these
as a list (Fig. 8(a)). When the user clicks a button to open
a specific story, files used by the current client (“Use by”
column in Table 2) are downloaded and loaded. When the
user uploads a story from the desktop plugin, all the files are
generated and uploaded to the server. The mobile client only
modifies msg.json and annImg i.png files.
When the user adds annotations on the mobile client,
we need to re-create video.mp4 and video backward.mp4 to
reflect the changes. For this purpose we integrated another
server-based application into the V-Mail infrastructure. This
application shares much functionality with the desktop client
and checks the database every minute for the changes. If it
finds any, it recreates the video files and uploads these to the
server. For this check we compare the last modified time by
a mobile client with that of a desktop in the database.
The mobile client plays video.mp4 or video backward.mp4
depending on the current play direction. When the user
clicks a backward playback button (Fig. 6) while watching
a video in a forward direction, the client skips time in
video backward.mp4 to match that in video.mp4 and plays
video backward.mp4. The same happens for video.mp4 when
the user clicks a forward playback button.
We built the desktop and mobile clients with Unity3D,
and the web client is a web page with a HTML video element.
We realized the database with MySQL, and wrote the
server-side scripts for accessing the database and download-
ing/uploading files in PHP. We used an FFmpeg executable
file for creating a video file from a series of images captured
from an exploratory visualization application [17].
5.2 Desktop and mobile clients details
We built the desktop and mobile clients using C# in Unity3D.
We handle storytelling and collaborative activities in V-
Mail with a StoryManager (Fig. 11). In addition to providing
features for loading and uploading a story to the server, the
StoryManager has three components for viewing and editing
a story. A Message manages content in the current story and
has an ordered list of Page(s). Each Page contains view and
Page
Contains infomration
about a single data view
Message
MessagePlayer
(Desktop only)
Computes the current
Transition based on the
time and Message
<<interface>>
IViewer
void SetState(Transition t)
MessagePlayer
(Mobile only)
Similar to the desktop
MessagePlayer yet the
time relies on a video
Exploratory Vis. Software
(Desktop only)
360° Images Viewer
(Mobile only)
Video Player
(Mobile only)
Transition
Contains two adjoining
Pages and the current
transition amount (0~1)
CommentViewer
AnnotationViewer
SphericalImageViewer
(Mobile only)
VisualizationViewer
(Desktop only)
1..*
Fig. 11. Three components of viewing/editing a story in StoryManager
visualization state information captured from an underlying
visualization application. It also contains comments and
sketches drawn by users. The MessagePlayer updates the
story time depending on the current playback option (Fig. 6).
It also provides methods for accessing the current Page and
Transition in a story. A Transition is composed of two Pages and
a transitioning amount. Viewers are visual components that
show the current Message and provide a means for editing
it. CommentViewer updates the comment panel (Fig. 2), and
AnnotationViewer shows drawings on the render view. The
VisualizationViewer updates an underlying visualization. If the
current time is on a page, the viewer updates the visualiza-
tion from the current Page accessed from the MessagePlayer. If
on a transition portion, the viewer interpolates between two
Pages from the Transition. As the mobile client plays video
files, the StoryManager in the client is different from that of
the desktop plugin. The VisualizationViewer is removed as the
client does not rely on an underlying visualization, and we
add a SphericalImageViewer to provide limited 3D exploration.
The MessagePlayer updates the timeline slider based on the
current time in the video, and skips time upon clicks on the
corresponding buttons or a specific point on the slider.
6 APPLICATION CASE AND EVALUATION
To evaluate the effectiveness of our concept for team collabo-
ration, we recruited members of a leading, interdisciplinary
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 10
climate science research team
to integrate V-Mail into their
normal workflow over a two-week period. This section
reports on training, observations, results, user feedback, and
other lessons learned from this real-world application. We
also showed the tool to members of a
neuroscience research
team
, and report lessons and user feedback from their initial
encounter with our tool in Section 6.3. Unlike the climate
science team, they decided not to be part of a longitudinal
study. In the section, we also report limitations and areas
could be addressed in future work.
6.1 Methodology
The interdisciplinary climate science research team uses a
combination of tools, including some of the world’s most
advanced high-performance computing (i.e., supercomputer)
simulations, to study complex interactions of Antarctic water
masses that may explain the global rise in sea temperature.
The team includes oceanographers, geologists, computer
scientists, a visual artist, and a science writer. Like many
research teams today, they place an increasing emphasis
on improved communication with the public. The artist and
writer have played leading roles in these efforts, and the team
has produced five short videos describing the science in the
past seven years, all of which were national competition
finalists (two won awards as best submissions). One of their
recent videos was created with the Unity-based Artifact-
Based Rendering (ABR) engine [17]—the same engine used
for our first implementation of a V-Mail plugin.
Since one of our goals was to understand how V-Mail
might integrate with whatever data communication work
is at hand in a real, research-active science team, we asked
the team to use V-Mail for whatever tasks they thought
made sense during the two-week trial we had agreed upon.
The team decided that the most interesting communication
task
at hand was to develop one or more complementary
videos to go deeper into certain aspects of the science and
provide useful material for the scientists to use in their
presentations. Thus, exploring how V-Mail might be utilized
to support this work became the primary task of the V-
Mail trial, and the
two primary users
became the visual
artist, a professional sculptor and painter for more than 30
years whose collaborations with scientists led her to data
visualization, and the science writer, a Ph.D. student in
cultural geography. During the first few days of the two-
week period, we met individually with the artist and science
writer to provide
training
for the V-Mail clients they chose to
use and to assist them in their tasks. We
recorded
qualitative
feedback by taking notes from interviews we conducted after
the training sessions as well as via questionnaires before and
after the two-week period. In addition, we saved each V-Mail
that they created and configured the V-Mail clients to log
all edits. After the two weeks, the artist shared the resulting
V-Mail in a regularly scheduled, “live” video-conference
with the
scientists
, gathering feedback on whether the story
covers important data features and whether it employs
appropriate storytelling techniques.
6.2 Observations and Results
We organize this section by the type of data we collected.
6.2.1 Pre-study Questionnaire
In the questionnaire we asked for the teams’ pre-existing
communications and related needs, and the two primary
participants filled out the questionnaire. Their responses
showed that they generally take different approaches for data
communication. For urgent matters, participants typically
communicate in-person or send e-mail-like messages. When
answering a question that could be answered in few words,
participants use an e-mail-like tool. However, when it takes
multiple steps to explain a concept, they typically use syn-
chronous communication, such as a live video conferencing
tool. The choice was the same for describing spatial features
in their data and for public-facing communication. When
asked about their use of visuals in e-mails, participants
responded differently. One does not use e-mails for sharing
3D visualization results. The other said that about 60% of her
e-mails contains images or videos, and 5% of these contain
annotations. With regards to device usage, both participants
typically have some access (2–8 hours) to high-end or
specialized devices. However, they typically use personal
PCs or mobile devices for science-related communications.
One participant indicated that her responses to questions
may vary depending on goals, as she often uses a high-end
PC when working on a video or visualization project.
6.2.2 Produced V-Mails
In total, the final V-Mail contains ten data views (Fig. 12). The
first three (#1–#3) provide an overview of data visualization,
highlighting all visible variables. The last shot (#10) is another
overview shot. Data views in the middle (#4–#9) describe two
different concepts about interactions of water masses and
currents on the ice shelf. Each concept was communicated
with three shots. The first (#4, #7) gives a spatial context, the
second (#5, #8) zooms in further to focus on visual decisions,
and the last (#6, #9) zooms in further to relate the visual
decisions to specific data features. Each shot is discussed in
detail with textual comments contributed by both artists.
Interested in sharing their work to collaborating scien-
tists and possibly using it as video footage for their next
video projects, the artist requested another version of the
V-Mail that shows the fly-through animation at a higher
resolution (1920
×
1080) and without the floating 3D pages.
Treating it like a final draft of their work, the “final” V-Mail
shows only the voice-over script, without all the low-level
communications such as questions in a form of comments.
6.2.3 Software Logs
Our log data and users’ comments made in the story indicate
that there were five back-and-forth communications over the
two weeks (Table 3). At each turn, after making edits, the
participants used their normal e-mail to pass the V-Mail link
to each other and request comments. Both URL and name
were used to reference updated V-Mails.
The textual comments were added to data views and
were used mainly to provide guidelines and script a voice-
over. Questions were also asked in a form of comments. For
example, confused with the scripting guidelines, the science
writer left questions on corresponding data views. We also
found that the comments contained imperfections, like typos
and this accidental use of caps lock, “hERE WE CAN SPEAK
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 11
Fig. 12. Visualizations of data views’ placements in the V-Mail participants created. Each circle represents a data view with a number indicating the
view’s order and with color indicating its thematic section.
TABLE 3
Five back-and-forth communications between participants.
# User Client Goals # of data # of edited # of views w/ # of views w/
views data views new comments new sketches
1 artist desktop capture data views, provide scripting guidelines 15 15 added 13 0
2 writer mobile draft a voice-over script, ask questions 15 none 9 0
3 artist desktop answer questions, make requests 15 none 3 0
4 writer mobile make requests 15 none 5 0
5 author desktop address the requests 10 5 removed, 0 0
1 rearranged
about.” Capitalization was also used for emphasis, as in
“IGNORE the above.” The artist and science writer did not
use the sketching feature to annotate data views.
The visual artist used the desktop client and the science
writer used the mobile client. While working on the script, the
science writer realized that a few data views were not needed.
While working on the mobile client, she left comments on
these extra data views to request a removal. Because the artist
using the desktop client forgot how to remove data views and
we (the authors) were also monitoring the communication,
we were able to quickly help out and resolve the request in
the desktop client.
6.2.4 Observations during introductory sessions
At the end of each training session, we asked users to now
begin to use the tool while we observed their actions and
answered any questions that came up.
In the first V-Mail,
the artist
captured data views with
scripting guidelines written as comments (Table 3). The
installation steps and training time for this stage took about
20 minutes for the desktop-client. In the next hour session,
she created three V-Mail drafts from scratch (from an empty
storyboard) and did not report any major difficulties. She
discarded two of them, and proceeded to refine to third by
adding a few more data views, rearranging their placement,
and adding notes and guidelines to send to her colleague.
The artist created V-Mails with multiple sections (i. e., data
views) to cover different data features (Fig. 12) and used
different approaches to extend and strengthen a story line
.
We noticed that the most recently added data view was often
used as a spatial reference frame for the next data view. For
example, after adding view #6, view #7 was described using
the language, “now looking down and reveal the currents.”
The participant also added data views “out of sequence” and
rearranged previously added views to strengthen the story.
In the second V-Mail,
the science writer
opened the
message from the artist, then proceeded to work on drafting
a voice-over script for each data view using the guidelines
contained in the V-Mail (Table 3). The science writer used the
mobile-client because it provided all features needed for her
tasks. The installation and training took her about 5 minutes.
She flipped through pre-existing data views and added the
script as comments, finishing this task in next 30 minutes.
She did not report any major difficulties while using the tool.
6.2.5 Feedback from User Interviews
After they created their first V-Mails, we interviewed the par-
ticipants to learn about their views of both tool and concept.
The artist described gaps in their current tools and
relevant cases where V-Mail could be most useful
. The
artist described her typical approach of creating a fly-through
animation as, first, establishing the concepts to convey and,
second, capturing the shots needed for the concepts. She
described the process as “melding story into visuals.” For
all but simple topics, in-person communication or video
conferencing are preferred for describing such visual and
spatial concepts, but in practice, they often resort to sending
several fly-though animations to the scientists to confirm the
concepts. In their traditional workflow, this process takes
a long time as they often require technical assistance from
additional team members to create their animations. V-Mails,
in contrast, enable artists, science writers, or other team
members to control this process on their own. They can focus
on identifying the right views to convey the concepts, and
the animation is automatically created. The ability to quickly
mock-up animations from data views expedites the creation
process, and the ability to view the results on mobile devices
and web browsers simplifies the distribution.
Imagining how their experiences might translate to
a
future trial with the larger group
, the artist pointed out that
the mobile client will provide a means for the scientists to
provide feedback without the need to go through complex
installation steps and a long training. She also pointed out
the importance of the web-client as it enables them to reach
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 12
people outside of the group, e. g., the public and scientists
from other groups. The artist did not use
the sketching
feature to annotate data views
, even though, in several data
views, she described features at specific locations of the views
(e. g., eddies formed in a current shown in the bottom). When
asked about it during the interview, the artist explained that
she thought this would be useful in some situations, but in
this case she knew the science writer was already familiar
with data features and visual mappings from their close
recent work together.
The participants made several
feature requests
. She re-
quested to implement additional V-Mail clients for ParaView
[2], a visualization tool the team frequently uses. The science
writer suggested supporting audio recording to annotate
existing data views as it is often easier than writing thoughts
down. The writer told us that opening a V-Mail takes a long
time on her device. She suggested another data presentation
mode that shows a V-Mail without the animated transitions,
as this can reduce the download time and the animated
transitions might not be needed for some users who are
already familiar with data presented in the shared views.
Also, in our current implementation, the user first needs to
add a data view before annotating it. She suggested another
approach where a data view is automatically added when
the user starts drawing on the current camera view.
6.2.6 Feedback from scientists
After the two weeks, the artist shared the resulting V-Mail in
a regularly scheduled, “live” video-conference with the larger
group, and led a discussion about the work using the web-
client. The scientists were most interested in seeing details
of water-mass distributions, current trails, and their impacts
on melting glaciers. The created V-Mail narrative covers
these regions of interests and employs proper storytelling
techniques—e. g., a series of first overview, then zoomed-in
data views [33] connected with transitioning animations—
to show not only details of these regions but also their
contextual information. They were also interested in seeing
changes of specific regions over time. Unfortunately, time-
series data is not yet supported in the underlying visualiza-
tion software. The scientists requested this as a future work
and brainstormed ideas for storytelling techniques to convey
these in future iterations. They envisioned to station the
camera to focus on a specific region and then show changes
over time. They would do this for multiple data regions
and employ V-Mail’s interactive storytelling techniques to
return to previous camera locations to facilitate comparative
analysis of the sites.
6.2.7 Post-study Questionnaire
One user from the climate science team participated in the
questionnaire. Her responses reaffirm what she described
about our tool in the interview session (Section 6.2.5), giving
positive responses on features that enable her to make movies
or tell data stories. However, she noted several additional
points. The participant noted that connecting views with
animations is a crucial component in data communication.
The ability to pause in-between and play backwards as well
as the ability to see the 3D frames, however, received neutral
responses. Nonetheless, she was concerned about using small
mobile displays as they may not be able to convey the visual
detail of the visualizations.
6.3 Initial Feedback from a Brain Science Team
We integrated V-Mail into a different system that visualizes
crossing pathways in white matter areas in a human brain
(see Appendix B). We showed the V-Mail during a live video
conference to members of a neuroscience research team,
who was already familiar with the visualization system.
Their initial reaction was somewhat mixed; they told us
the tool itself is novel and could be useful in presenting
findings to a broader audience, but there is no urgent need
for such tool for their team-science collaboration. The team
already meets regularly, almost weekly, using a live video
conference or in a meeting room, and the team is led by
one senior member to whom his students report. They use
remote asynchronous communication tools such as e-mails
to coordinate the meetings or share datasets.
Participants described situations where cross-platform
collaboration tools like V-Mail may not work and could be
useful. In clinics, radiologists use specialized monitors for
viewing lossless DICOM images to see subtle differences in
tissue contrasts. Also a participant pointed out that there is a
limit to what information they can share with the public as
certain information is sensitive and private. In cases where
sharing information does not have such constraints, V-Mails
could be used to enhance the credibility of presented results.
They envision that stories created in V-Mail are submitted
as supplementary material for research papers. Linked from
figures or text, V-Mails could allow reviewers or readers to
validate the authors’ claims and further explore the data.
6.4 Discussion, Limitations, and Future Work
The success of adding a cross-device, asynchronous commu-
nication tool, like V-Mail, into the existing science team’s
workflow depends greatly on how the team already works.
The brain science team decided not to partake in a two-week
trial of V-Mail, as this team included just one senior member
and several grad students who already meet regularly to
discuss their work. They also raised concerns about security
with respect to patient data that should not be widely
shared (Section 6.3). The climate science team also expressed
a preference for synchronous communication. However,
schedules are more complex within this larger team, so the
ability to communicate asynchronously is more valuable.
In addition to using the comment feature for drafting a
voice-over script, participants used it to ask questions and
delegate tasks, and these comments often contain typos and
accidental use of caps lock. While these low-level communi-
cations were removed from the “final” V-Mail, they can help
people to clarify tasks and enable other team members to
provide immediate help. All of these reinforce the chat-like
immediacy and draft-in-progress qualities of V-Mail. These
qualities are unusual in science communication, especially
high-end data visualization, but we believe they may, in fact,
help to put a human face on science communication, helping
scientists and stakeholders more freely discuss scientific data.
The science writer pointed out that communicating data
in a form of stories might introduce unnecessary overhead for
audiences who are already familiar with data (Section 6.2.5).
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 13
For these users, the ability to disable certain storytelling
features and share only portions of a data story could be
useful. By adapting the feature that encodes all viewing
properties in a shared link [29], we could associate an URL
parameter to each data view in a story, so that a specific
view could be referenced in e-mails or comments. The
feature would also be required if V-Mail is used with other
storytelling media, e. g., for providing interactive figures in a
paper submission as discussed by the brain science team. In
the same spirit, other storytelling paradigms that rely less on
animations could also be integrated in V-Mail. For instance,
scrolly-telling [34] can be used to rapidly control the speed
and direction of viewing. We could also spread out data
views in a layout, e. g., as done in data comics [5], which
could be more useful for audiences who are already familiar
with spatial locations of data features.
A major limitation of the current V-Mail implementation
is that it can only be integrated into applications built using
Unity3D, and a separate instance of the V-Mail plug-in is
needed for each system. To ease the integration process,
we are considering a more loosely coupled implementation
where a separate storyboard application communicates over
a network with visualization software. Since the webpage
and the mobile app do not rely on other data visualization
applications (these clients play a video), we can continue
using these clients for showing and interacting with stories
created from new connected software.
We observed that participants used lightweight clients if
the features required for their tasks were available in those
clients. We wonder, if the clients supported some lightweight
data exploration features, participants might have explored
data even though their tasks did not require this. Indeed, the
brain science team viewed facilitating interactive exploration
on the web-client as an important direction for future work.
Thus, upgrading the web-client with remote rendering [26] or
new web technologies [38] could be a useful next direction. In
addition, as mentioned by the artist and the brain scientists,
seeing visualization on small screens, like a phone, is just
limiting. Further studies are needed to better characterize
exactly how much value the capability provides to science
teams. In addition, further studies with more and other
domain experts should be carried out to confirm our findings
from our qualitative evaluation.
7 CONCLUSION
On a conceptual level, our work extends the state of art
on storytelling in visualization and provides a rich tool
set for asynchronous remote collaboration. We build upon
existing spatial 3D data storytelling techniques that tightly
couple storytelling with data exploration and employ semi-
automatic procedures to expedite the story creation process.
What is unique in V-Mail is the ability to collectively work
on data stories, which is often seen with abstract data
visualization, e. g., via web-based visualizations, but less so
with visualization of spatial 3D data. Essentially this multi-
author editing in V-Mail is a type of parallel storytelling,
with the difference that users not only view or edit shared
data stories but also do this across different devices. V-Mail
is thus an example of using multiple display devices in
collaborative data visualization [15], [16]. V-Mail focuses,
however, on asynchronous remote communication and pro-
vides lightweight mobile and web clients for partaking in
collaboration and simplifies the distribution process using
URLs to reference data stories.
The desktop-client is an example of asynchronous col-
laboration frameworks that support collaborative activities
around exploratory visualizations. The mobile-client, how-
ever, supports these activities around video-based visualiza-
tions created from the desktop-client. The ability to provide
visual feedback directly onto shared visualization results is
similar to the ability to annotate on shared data views and
stories in the web-based visualization. However, there are
two major differences in our approach: (1) communication
mechanisms are tailored for spatial 3D data visualization,
e. g., with spatial annotations and animated transitions,
and (2) the app is designed to solely work with shared
visualization results. The latter point may be of interest to
visualization practitioners as this approach enables them
to bypass installation steps and computation requirements
often imposed by using exploratory visualization software
such as ParaView [2]. Also this separation suggests an inter-
esting future direction of turning V-Mail into a collaborative
framework that unifies well-known software in the field for
creating data stories and conducting discussion.
In our first V-Mail deployment and evaluation, we took
the approach of asking a real science team to integrate V-Mail
in whatever ways they found most useful for the tasks at
hand during a two-week period. The results were generally
positive, re-confirming the urgent need for improved tools for
spatial communication in this context, while also suggesting
areas of required improvements to advance V-Mail from
the prototype stage to a production-level tool. In this first
application, only the artist and science writer actively used
the tool to draft a video-based visualization, and the scientists
were involved at the last stage to provide feedback on the
created visualization. The brain science team provided initial
feedback on the tool but did not partake in a longitude
study. Notably, if we are able to secure funding to support
additional work on V-Mail, we plan follow-on studies to
understand how V-Mail might be used and/or tailored to
best support specific scientist-to-scientist, public-to-science-
team, and lay-person-to-specialist communications.
ACKNOWLEDGMENTS
The authors thank Seth Johnson and Bridger Herman for
assistance with integrating the ABR engine. This research
was supported in part by the National Science Foundation
(IIS-1704604, IIS-1704904).
IMAGES/FIGURES LICENSE AND COPYRIGHT
We as authors state that all of our figures in this article are
and remain under our own personal copyright, with the
permission to be used here. We also make them available
under the Creative Commons Attribution 4.0 International
(c b CC BY 4.0) license and share them at osf.io/qehvs.
REFERENCES
[1]
D. Acevedo, E. Vote, D. H. Laidlaw, and M. S. Joukowsky,
“Archaeological data visualization in VR: Analysis of lamp finds at
the Great Temple of Petra, a case study,” in Proc. Visualization. Los
Alamitos: IEEE CS, 2001, pp. 493–496. doi:
10.1109/VISUAL.2001.964560
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 14
[2]
J. Ahrens, B. Geveci, and C. Law, “Paraview: An end-user tool
for large data visualization,” in The Visualization Handbook, C. D.
Hansen and C. R. Johnson, Eds. Oxford: Elsevier, 2005, ch. 36, pp.
717–731. doi: 10.1016/B978-012387582-2/50038-1
[3]
H. Akiba, C. Wang, and K.-L. Ma, “AniViz: A template-based
animation tool for volume visualization,” IEEE Comput Graph Appl,
vol. 30, no. 5, pp. 61–71, 2010. doi: 10.1109/MCG.2009.107
[4]
B. Bach, N. Kerracher, K. W. Hall, S. Carpendale, J. Kennedy, and
N. Henry Riche, “Telling stories about dynamic networks with
graph comics,” in Proc. CHI. New York: ACM, 2016, pp. 3670–
3682. doi: 10.1145/2858036.2858387
[5]
B. Bach, Z. Wang, M. Farinella, D. Murray-Rust, and
N. Henry Riche, “Design patterns for data comics,” in Proc. CHI.
New York: ACM, 2018, pp. 38:1–38:12. doi: 10 .1145/3173574.3173612
[6]
K. Bladin, E. Axelsson, E. Broberg, C. Emmart, P. Ljung, A. Bock,
and A. Ynnerman, “Globe browsing: Contextualized spatio-
temporal planetary surface visualization,” IEEE Trans Vis Comput
Graph, vol. 24, no. 1, pp. 802–811, 2018. doi:
10.1109/TVCG.2017.2743958
[7]
P. de Lange, P. Nicolaescu, J. Benscheid, and R. Klamma, “Collab-
orative non-linear storytelling around 3D objects,” in Proc. ICWL.
Cham: Springer, 2018, pp. 88–98. doi: 10 .1007/978-3-319-96565-9 9
[8]
N. Duarte, Data Story: Explain Data and Inspire Action Through Story.
Ideapress Publishing, 2019.
[9]
R. Eccles, T. Kapler, R. Harper, and W. Wright, “Stories in geotime,”
Inf Vis, vol. 7, no. 1, pp. 3–17, 2008. doi: 10.1057/palgrave.ivs.9500173
[10]
N. Gershon and W. Page, “What storytelling can do for information
visualization,” Commun ACM, vol. 44, no. 8, pp. 31–37, 2001. doi:
10.1145/381641.381653
[11]
J. Heer and M. Agrawala, “Design considerations for collaborative
visual analytics,” Inf Vis, vol. 7, no. 1, pp. 49–62, 2008. doi:
10.
1057/palgrave.ivs.9500167
[12]
J. Heer and G. Robertson, “Animated transitions in statistical data
graphics,” IEEE Trans Vis Comput Graph, vol. 13, no. 6, pp. 1240–
1247, 2007. doi: 10.1109/TVCG.2007.70539
[13]
J. Heer, F. B. Vi
´
egas, and M. Wattenberg, “Voyagers and voyeurs:
supporting asynchronous collaborative information visualization,”
in Proc. CHI. New York: ACM, 2007, pp. 1029–1038. doi:
10.
1145/1240624.1240781
[14]
S. Iserhardt-Bauer, P. Hastreiter, T. Ertl, K. Eberhardt, and
B. Tomandl, “Case study: Medical Web service for the automatic 3D
documentation for neuroradiological diagnosis,” in Proc. Visualiza-
tion. Los Alamitos: IEEE CS, 2001, pp. 425–581. doi:
10.1109/VISUAL.
2001.964542
[15]
D. H. Jeong, S.-Y. Ji, E. A. Suma, B. Yu, and R. Chang, “Designing a
collaborative visual analytics system to support users’ continuous
analytical processes,” Hum-Centric Comput Inf Sci, vol. 5, no. 1, pp.
5:1–5:20, 2015. doi: 10.1186/s13673-015-0023-4
[16]
D. H. Jeong, E. Suma, T. Butkiewicz, W. Ribarsky, and R. Chang,
“A continuous analysis process between desktop and collaborative
visual analytics environments,” in Proc. VAST. Los Alamitos: IEEE
CS, 2010, pp. 231–232. doi: 10.1109/VAST.2010.5652958
[17]
S. Johnson, F. Samsel, G. Abram, D. Olson, A. J. Solis, B. Herman,
P. J. Wolfram, C. Lenglet, and D. F. Keefe, “Artifact-based rendering:
Harnessing natural and traditional visual media for more expres-
sive and engaging 3D visualizations,” IEEE Trans Vis Comput Graph,
vol. 26, no. 1, pp. 492–502, 2019. doi: 10.1109/TVCG.2019.2934260
[18]
T. Klein, L. Autin, B. Kozl
´
ıkov
´
a, D. S. Goodsell, A. Olson, M. E.
Gr
¨
oller, and I. Viola, “Instant construction and visualization of
crowded biological environments,” IEEE Trans Vis Comput Graph,
vol. 24, no. 1, pp. 862–872, 2017. doi: 10.1109/TVCG.2017.2744258
[19]
C. N. Knaflic, Storytelling with Data: A Data Visualization Guide for
business Professionals. Hoboken: Wiley, 2015.
[20]
R. Kosara and J. Mackinlay, “Storytelling: The next step for
visualization,” Comput, vol. 46, no. 5, pp. 44–50, 2013. doi:
10.1109/MC.
2013.36
[21]
D. Kou
ˇ
ril, O. Strnad, P. Mindek, S. Halladjian, T. Isenberg, M. E.
Gr
¨
oller, and I. Viola, “Molecumentary: Adaptable narrated docu-
mentaries using molecular visualization,” IEEE Trans Vis Comput
Graph, 2022, to appear. doi: 10.1109/TVCG.2021.3130670
[22]
B. Lee, R. H. Kazi, and G. Smith, “SketchStory: Telling more
engaging stories with data through freeform sketching,” IEEE
Trans Vis Comput Graph, vol. 19, no. 12, pp. 2416–2425, 2013. doi:
10.
1109/TVCG.2013.191
[23]
B. Lee, N. H. Riche, P. Isenberg, and S. Carpendale, “More than
telling a story: Transforming data into visually shared stories,”
IEEE Comput Graph Appl, vol. 35, no. 5, pp. 84–90, 2015. doi:
10.
1109/MCG.2015.99
[24]
I. Liao, W.-H. Hsu, and K.-L. Ma, “Storytelling via navigation: A
novel approach to animation for scientific visualization,” in Proc.
Smart Graphics. Cham: Springer, 2014, pp. 1–14. doi:
10.1007/978-3
-319-11650-1 1
[25]
E. M. Lidal, M. Natali, D. Patel, H. Hauser, and I. Viola, “Geological
storytelling,” Comput Graph, vol. 37, no. 5, pp. 445–459, 2013. doi:
10.1016/j.cag.2013.01.010
[26]
E. J. Luke and C. D. Hansen, “Semotus Visum: A flexible remote
visualization framework,” in Proc. Visualization. Los Alamitos:
IEEE CS, 2002, pp. 61–68. doi: 10.1109/VISUAL.2002.1183758
[27]
P. Lundblad and M. Jern, “Geovisual analytics and storytelling
using HTML5,” in Proc. IV. Los Alamitos: IEEE CS, 2013, pp.
263–271. doi: 10.1109/IV.2013.35
[28]
K.-L. Ma, I. Liao, J. Frazier, H. Hauser, and H.-N. Kostis, “Scientific
storytelling using visualization,” IEEE Comput Graph Appl, vol. 32,
no. 1, pp. 12–19, 2012. doi: 10.1109/MCG.2012.24
[29]
J. Maitin-Shepard, A. Baden, W. Silversmith, E. Perlman, F. Collman,
T. Blakely, J. Funke, C. Jordan, B. Falk, N. Kemnitz, tingzhao,
C. Roat, M. Castro, S. Jagannathan, moenigin, J. Clements, A. Hoag,
B. Katz, D. Parsons, J. Wu, L. Kamentsky, P. Chervakov, P. Hubbard,
S. Berg, J. Hoffer, A. Halageri, C. Machacek, K. Mader, L. Roeder,
and P. H. Li, “google/neuroglancer:,” Software: WebGL-based
viewer for volumetric data, 2021. doi: 10.5281/zenodo.5573294
[30]
M. Meuschke, L. Garrison, N. Smit, S. Bruckner, K. Lawonn, and
B. Preim, “Towards narrative medical visualization,” arXiv preprint
2108.05462, 2021.
[31]
N. H. Riche, C. Hurter, N. Diakopoulos, and S. Carpendale, Data-
Driven Storytelling. Boca Raton: CRC Press, 2018. doi:
10.1201/
9781315281575
[32]
E. Segel and J. Heer, “Narrative visualization: Telling stories with
data,” IEEE Trans Vis Comput Graph, vol. 16, no. 6, pp. 1139–1148,
2010. doi: 10.1109/TVCG.2010.179
[33]
B. Shneiderman, “The eyes have it: A task by data type taxonomy
for information visualizations,” in Proc. VL. Los Alamitos: IEEE
CS, 1996, pp. 336–343. doi: 10.1109/VL.1996.545307
[34]
C. D. Stolper, B. Lee, N. H. Riche, and J. Stasko, “Emerging and
recurring data-driven storytelling techniques: Analysis of a curated
collection of recent stories,” Microsoft Research, Washington, Tech.
Rep. MSR-TR-2016-14, 2016.
[35]
M. Th
¨
ony, R. Schn
¨
urer, R. Sieber, L. Hurni, and R. Pajarola,
“Storytelling in interactive 3D geographic visualization systems,”
ISPRS Int J Geo-Inf , vol. 7, no. 3, pp. 123:1–123:14, 2018. doi:
10.
3390/ijgi7030123
[36]
C. Tong, R. Roberts, R. Borgo, S. Walton, R. Laramee, K. Wegba,
A. Lu, Y. Wang, H. Qu, Q. Luo et al., “Storytelling and visualization:
An extended survey,” Inf , vol. 9, no. 3, pp. 65:1–65:42, 2018. doi:
10.
3390/info9030065
[37]
M. Tsang, G. W. Fitzmaurice, G. Kurtenbach, A. Khan, and B. Bux-
ton, “Boom Chameleon: Simultaneous capture of 3D viewpoint,
voice and gesture annotations on a spatially-aware display,” in
Proc. UIST. New York: ACM, 2002, pp. 111–120. doi:
10.1145/571985.
572001
[38]
W. Usher and V. Pascucci, “Interactive visualization of terascale
data in the browser: Fact or fiction?” in Proc. LDAV. Los Alamitos:
IEEE CS, 2020, pp. 27–36.
[39]
F. B. Viegas, M. Wattenberg, F. Van Ham, J. Kriss, and M. McKeon,
“ManyEyes: A site for visualization at internet scale,” IEEE Trans Vis
Comput Graph, vol. 13, no. 6, pp. 1121–1128, 2007. doi:
10.1109/TVCG.
2007.70577
[40]
M. Wohlfart and H. Hauser, “Story telling for presentation in
volume visualization,” in Proc. EuroVis. Goslar: Eurographics
Assoc, 2007, pp. 91–98. doi: 10.2312/VisSym/EuroVis07/091-098
[41]
K. Wongsuphasawat, D. Smilkov, J. Wexler, J. Wilson, D. Mane,
D. Fritz, D. Krishnan, F. B. Vi
´
egas, and M. Wattenberg, “Visualizing
dataflow graphs of deep learning models in tensorflow,” IEEE Trans
Vis Comput Graph, vol. 24, no. 1, pp. 1–12, 2017. doi:
10.1109/TVCG.
2017.2744878
[42]
A. Ynnerman, J. L
¨
owgren, and L. Tibell, “Exploranation: A new
science communication paradigm,” IEEE Comput Graph Appl,
vol. 38, no. 3, pp. 13–20, 2018. doi: 10.1109/MCG.2018.032421649
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.
SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 15
Jung Who Nam
is a doctoral student at the Uni-
versity of Minnesota, Twin Cities. His research in-
terests include scientific visualization, immersive
analytics, and data storytelling. He is particularly
interested in novel visualization and interaction
methods that support exploration and collabora-
tion around spatial 3D data visualization.
Tobias Isenberg
is a senior research scientist
at Inria, France. Previously he held positions
as post-doctoral fellow at the University of Cal-
gary, Canada, and as assistant professor at the
University of Groningen, the Netherlands. His
research interests include scientific visualiza-
tion, illustrative and non-photorealistic rendering,
and interactive visualization techniques. He is
particularly interested in interactive visualization
environments for 3D spatial data.
Daniel F. Keefe
is a Distinguished University
Teaching Professor and Professor of Computer
Science and Engineering at the University of
Minnesota. His research centers on scientific
data visualization and interactive computer graph-
ics. In addition to his work in computer science,
Keefe is also an accomplished artist and has
published and exhibited work in top international
venues for digital art. Before joining the University
of Minnesota, Keefe did post-doctoral work at
Brown University jointly with the departments of
Computer Science and Ecology and Evolutionary Biology and with the
Rhode Island School of Design. He received the Ph.D. in 2007 from
Brown University’s Department of Computer Science and the B.S. in
Computer Engineering summa cum laude from Tufts University in 1999.
This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.